Summary

• val_bpb = 1.02045733 (3-seed mean, std 0.00553017) | 15.31–15.83 MB | 8×H100 SXM
• derived from the GDN-Hybrid line of Record: GDN-Hybrid (Gated DeltaNet + SWA) — val_bpb 1.028308 (3-seed cold-cache mean) #1545

3-Seed Results

Architecture

This submission uses an SP1024-tokenized GDN-Hybrid backbone with the following high-level structure:

[GDN×5] → SWA → [GDN×5] → SWA_shared

Key components:

1. SP1024 tokenizer
2. Gated DeltaNet hybrid backbone
3. Sliding-window attention side path
4. MuonEq-R + AdamW training mix
5. EMA = 0.997
6. Late QAT threshold = 0.15
7. GPTQ int6 + zstd-22 packaging

Credits

• PR Record: GDN-Hybrid (Gated DeltaNet + SWA) — val_bpb 1.028308 (3-seed cold-cache mean) #1545 — GDN-Hybrid architecture line ([GDN×5] → SWA → [GDN×5] → SWA_shared)
• PR Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean) #1019 — prior SP1024 / no-TTT / quantized record-track lineage
• PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 — earlier record-track baseline